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I'lem or invenuon \ 

This invention relates generally to computer software applications and 
specifically to testing computV software applications. 



Background 

Distributed computing ha^been used for many years. Distributed computing is 
very prevalently used in "enterprisVwide" applications. An enterprise-wide application 
is an application that allows a large Wup of people to work together on a common task. 
Usually, an enterprise-wide application performs fimctions that are essential to a 
company's business. For example, inV bank, people at every bank branch must be able 
to access a database of accounts for eVfery bank customer. Likewise, at an insurance 
company, people all over the company Viust be able to access a database containing 
information about every policyholder. tKc software that performs these fimctions is 
generally known as enterprise-wide applicatiims. 



A available hardware and software has evolved, the architecture of enterprise 
wide applications has changed. An architecture which is currently popular is called the 
N-Tier entebrise model. The most prevalent N-tier enterprise model is a three tier 
model. The three tiers are the front end, the middleware and the back end. The back end 
is the database\ The front end is sometimes referred to as a "client" or a Graphical User 
Interface (GUI)\ The middleware is the software that manages interactions with the 
database and ca^es the "business logic." Business logic tells the system how to 
validate, process aM report on the data in a fashion that is usefiil for the people in the 
enterprise. 

The middleware fesides on a computer called a server. The database might be on 
the same computer or a different computer. The "client" is usually on an individual's 
personal computer. All ofVhe computers are connected together through a network. 
Because many people use tHe enterprise wide application, such systems are set up to 
allow simultaneous users and tiaere would be many clients connected to a single server. 
Often, many clients will be connected to the server simultaneously. 

Those familiar with internet tommerce will recognize that the N-tiered model also 
describes many internet web sites thM sell goods or services. For example, a web site 
that auctions cars is likely to fit the N-tkred model. In such an application, databases are 
provided to track buyers, sellers and objects being auctioned. Also, a database must be 
provided to track the bids as they are enWed. The middleware provides the access to 
these databases and encapsulates the business logic around such transactions as when to 
accept a bid, when to declare an item sold, e\c. In the world of distributed computing, it 
makes no difference whether the "clients" usiik the application are employees of a single 
company or many Internet users throughout thd worid. Herein, examples of applications 
under test will be given, but they are not intended to imply limitations on the use of the 
invention. The inventions described herein coul\ be used by developers of enterprise- 
wide applications or web based applications. 

One advancement in the N-tiered model is that\he middleware is very likely to be 
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;ompbnentized and is very likely to be written to a component standard so that it will 
easily Integrate witU software at other tiers. Enterprise JavaBeans (EJB) by Sun 
Microsystems, COM, DCOM and COM+ by Microsoft Corporation and CORBA by IBM 
are exampWs of component specification standards that are contmiercially available. 
Herein, EJBVis used as an example of a component standard used to implement 
middleware in\an N-tiered model, but it should be appreciated that and the concepts 
described hereiX could be used with other component standards. These software 
components are t^hnology based software components. An EJB technology based 
software component Vill be referred to herein as simply an EJB component. 

EJB componentsNare written in the JAVA language, which is intended to be 
"platform independent." Platform independent means that an application is intended to 
perform the same regardless of the hardware and operating system on which it is 
operating. Platform independdnce is achieved through the use of a "container." A 
container is software that is desWed for a specific platform. It provides a standardized 
environment that ensures the ap^cation written in the platform independent language 
operates correctly. The container\s usually commercially available software and the 
application developer will buy the container rather create it. 

Componentized software is softie that is designed to allow different pieces of 
the application, or "objects", to be created, separately but still to have the objects work 
together. For this to happen, the objectsVnust have standard interfaces that can be 
understood and accessed by other objects. SoVe parts of these interfaces are enforced by 
the software language. If the interfaces are notWd, the software objects will not be able 
to work with other objects. Other practices areVmposed by convention. Usually, one 
company has "control" over the language and Wcifies programming practices that 
should be followed by anyone writing platform independent software in that language. 
Because these programming practices are known to eVeryone, the companies that create 
the containers can rely on them when creating the Wtainer. As a result, if these 
practices are not followed, the container might not operate properly. Thus, there is an 
indirect mechanism for enforcing these practices. 
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Typically, applications have been tested in one of two ways. The objects are 
tested asUhey are written. Each is tested to ensure that it performs the intended function. 
When thd objects are assembled into a completed application, the entire application is 
then usualW tested. Heretofore, application testing has generally been done by applying 
test inputs \t the cHent end and observing the response of the application. There are 
several shoitfcomings with this process. One is that it is relatively labor intensive, 
particularly to develop a load or scalability test. There has been no easy way to create 
the test progranAinstantiate it with test data, execute the test and aggregate the results. 

Some toolsXcalled "profilers," have been available. However, these tools track 
things such as disk i^age, memory usage or thread usage of the application under test. 
They do not provide daka about performance of the application based on load. 

Other tools are av^able to automate the execution of tests on applications. For 
example, RSW Software, InV of Waltham, MA, provides a product called e-Load. This 
tool simulates load on an application under test and provides information about the 
performance of the applicationX However, this tool does not provide information about 
the components in an applicationX We have recognized that a software developer would 
find such information very usefiil. \ 

Automatic test generation todls, such as TestMaster available from Teradyne 
Software and System Test of Nashua, NM, are also available. Tools of this type provide a 
means to reduce the manual effort of generating a test. TestMaster works from a state 
model of the application under test. Suc\ an application is very usefial for generating 
fimctional tests during the development okan application. Once the model of the 
application is specified, TestMaster can be insLcted to generate a suite of tests that can 
be tailored for a particular task - such as to fiillyVxercise some portion of the application 
that has been changed. Model based testing is particularly usefiil for fimctional testing of 
large applications, but is not fiilly automatic becaise it requires the creation of a state 
model of the application being tested. \ 
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We have recognized that a second shortcoming of testing enterprise wide 
applications is the critical performance criteria to measure often relates to how the 
applicatioA behaves as the number of simultaneous users increases. There are examples 
of websites Washing or operating so slow as to frustrate an ordinary user when too many 
users log onVimultaneously. In the past, load has been simulated informally, such as by 
having several people try to use the application at the same time. Some tools exist to 
provide a loaAon an application for testing, such as e-Load available from RSW of 
Waltham, MA. \ 

However, i\ has generally not been until the application is deployed into its 
intended operating ^vironment that the performance of the application under load is 
known. Thus, the biggest problem facing an application developer might not be testing to 
see whether each object\erforms as designed or even whether the objects work together 
as a system. Heretofore there has been no available tool that will help an application 
developer ascertain how fnany simultaneous users a middleware application can 
accommodate given a specified transaction response time or identify which object in the 
application, given real world load conditions, is causing the bottleneck. 

SUMMARY OF THE INVENTION 

With the foregoing backgrourid in mind, it is an object of the invention to provide 
testing tools to facilitate load based testing of N-tiered applications. It is also an object to 
provide automatic testing. The foregoinkand other objects are achieved by a test system 
that simulates use of a particular software\object within an application by a plurality of 
simultaneous users. The number of simuWieous users simulated is varied. In the 
preferred embodiment, load testing is done onVidividual components in the application. 

In a presently preferred embodiments, tke test system analyzes response time 
measurements from plural software objects withik the application and predicts which 
software object within the application that is likely toVe a performance bottleneck. 
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/ In other presently preferred embodiments, performance settings within the 
Jpphcation ^ also be varied to determine optimum settings to reduce performance 
bottlenecks. \ 

In still oth^r preferred embodiments, the format of the output may be specified by 
the user to aid in understanding the performance of the application under test in response 
to load. \ 

BRIEF DESCRIPTION THE DRAWINGS 

The invention willVbe better understood by reference to the foUov^^ing more 
detailed description and accompanying drawings in which 

FIG. 1- is an illustration of an application under test by the test system of thde 
invention; \ 

FIG. 2 is an illustration shoVing the test system of the invention in greater detail; 
FIG. 3 is an illustration showW the coordinator of FIG. 2 in greater detail; 
FIG. 4 is a flow chart illustrattng the process of coordinating execution of load 

tests; \ 
FIG. 5 is an illustration showing th\code generator of FIG. 2 in greater detail; 
FIG. 6 illustrates a user interface of t6st system 110 during a setup phase; 
FIG. 7 illustrates a user interface of te^ system 110 during the specification of a 

test case; \ 
FIG. 8 illustrates a user interface of test sVstem 110 during a different action as 

part of the specification of a test case\ 
FIG. 9 illustrates a user interface of test systdm 110 during a different action as 

part of the specification of a test case; \ 
FIG. 10 illustrates a user interface of test system\l0 during a different action as 

part of the specification of a test case; \ 
FIG. 1 1 illustrates a user interface of test system 1 loVuring a phase for reviewing 

the results of a test case in tabular format; \ 
FIG. 12 illustrates a user interface of test system 1 10 diMng a phase for reviewing 
the results of a test case in graphical format; \ 
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FIG. rs illustrates a user interface of test system 110 during a phase for reviewing 
' «le results of a test case in an alternative graphical format; and 

FIG. 14 illVgtrates a user interface of test system 110 during a phase for reviewing 
the relsults of a test case in a tabular format. 

descriptionVf the preferred embodiment 

FIG. 1 illustratesNa test system 110 according to the present invention. The 
system is testing applicatiWi under test 114. Here application under test 114 is an 
application in the N-tieredVodel. More specifically, it is a three tiered database 
application. AppUcation undektest 114 could represent a database for a bank or an 
insurance company or it could re^tesent an Internet application. The specific fimction of 
application under test 1 14 are not inlportant to the invention. 

Also, the specific hardware onVhich test system 110 and the application under 
test 114 reside is not important to thdV invention. It is sufficient if there is some 
connection between the two. In the illusl)lition, that connection is provided by network 
122. In the illustrated embodiment, it is con^mplated that network 122 is part of a LAN 
operated by the owner of application under te^l 14, such as an Ethernet network. In this 
scenario, test system 1 10 is installed on a serve\ within the LAN. However, many other 
implementations are possible. For example, netwWk 122 could be a WAN owned by the 
owner of application under test 114. \ 

A fiirther variation is that network 122 could tl^ Internet. In that scenario, test 
system 1 10 could be located in a server owned by a testing company. 

A fiirther possibility is that test system and applicatioV under test could be located 
in computers owned by a testing company. Many applicationW written using platform 
independent technology such that the application will perfWi the same on many 
different platforms. Platform independent technology is intended to make it easy to run 
an application on any platform. Therefore, the application underVest could be sent to a 
testing company, owning tiie hardware to implement test systkn 110, such as by 
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3loa(«hg over the Internet. Thereafter, the application under test could be tested as 
dlscribedk herein while running on a platform provided by the testing company with the 
results of tke test being downloaded over the Internet. 

Still otker variations are possible. Test system 1 10 and application under test 1 14 
could physicaliy^e implemented in the same computer. However, that implementation is 
not presently preyed because a single computer would have to be very large or would 
be limited in the site of applications that could be tested. The presently preferred 
embodiment uses seveWil computers to implement test system 1 10. 

Application underVest 114 is a software application as known in the art. It 
includes middleware 116 tHat encapsulates some business logic. A user accesses the 
application through a client dWice. Many types of client devices are possible, with the 
list growing as networks becom\more prevalent. Personal computers, telephone systems 
and even household appliances W^th micro- controllers could be the client device. For 
simplicity, the client device is illusttated herein as a personal computer (PC) 120, though 
the specific type of client device is ndt important to the invention. PC 120 is connected 
to network 122 and can therefore ac^ss application under test 114. In use, it is 
contemplated that there would be multipldaisers connected to application under test 114, 
but only one user is shown for simplicity. The number of users simultaneously accessing 
application under test 1 14 is one indication of the "load" on the application. 

Access to the application under test is, ik the illustrated embodiment, through 
Graphical User Interface (GUI) 124 of the type knbwn in the art. Software to manage 
interactions between multiple users and an applic^on is known. Such software is 
sometimes called a web server. Web servers operate in conjunction with a browser, 
which is software commonly found on most PC's. \ 

The web server and browser exchange information ill a standard format known as 
HTML. An HTML file contains tags, with specific informatiVi associated with each tag. 
The tag signals to the browser a type associated with the infoVnation, which allows the 
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browseAp display the information in an appropriate format. For example, the tag might 
signal' whVer the information is a title for the page or whether the information is a link 
to another Veb page. The browser creates a screen display in a particular window 
running on PQ 120 based on one or more HTML pages sent by the web server. 

When a \ser inputs commands or data into the window of the browser, the 
browser uses the irtformation on the HTML page to format this information and send it to 
the web server. InViis way, the web server knows how to process the commands and 
data that comes from uoe user. 

GUI 124 passes thfe information and commands it receives on to middleware 1 16. 
In the example of FIG. \ middleware 116 is depicted as a middleware application 
created with EJB components. Containers 130 are, in a preferred embodiment, 
commercially available contairters. Within a container, are numerous enterprise Java 
beans 132. Each Java bean can rftore generally be thought of as a component. GUI 124, 
based on the information from usdr PC 120, passes the information to the appropriate 
EJB component 132. Outputs from a^lication under test 1 14 are provided back through 
GUI 124 to PC 120 for display to a useX 

EJB component's 132, in the ilWistrated example, collectively implement a 
database application. EJB component's 132Yanage interactions with and process data 
from databases 126 and 128. They will perforrfl such database functions as setting values 
in a particular record or getting values from aVarticular record. Other ftmctions are 
creating rows in the database and finding rows irUie database. EJB component's that 
access the database are often referred to as "entity beans." 

Other types of EJB component's perform cWputation or control ftmctions. 
These are called \ 

"session beans." Session beans perform such ftinction^as validating data entries or 
reporting to a user that an entry is erroneous. Session beansVenerally call entity beans to 
perform database access. \ 
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It AjU be appreciated that, while it is generally preferable to segregate 
progsamminkof the application in such a way that each type of database transaction is 
controlled by V single bean that performs only that function, some entity beans will 
perform functioVs not strictly tied to database access. Likewise, some session beans will 
perform databasdaccess functions without callmg an entity bean. Thus, while different 
testing techniques Vill be described herein for testing session beans and entity beans, it is 
possible that some BJB component's will have attributes of both entity and session beans. 
Consequently, a ful\test of any bean might employ techniques of testing entity beans 
and testing session bee 

Test system 110 isVble to access the EJB component's 132 of application under 
test 1 14 over network 122. \n this way, each bean can be exercised for testing. In the 
preferred embodiment, the testis are predominately directed at determining the response 
time of the beans - or more geWUy determining the response time of components or 
objects used to create the application under test. Knowing the response time of a bean 
can allow conclusions about the perfbrmance of an application. The details of test system 
1 10 are described below. 

In the illustrated embodiment, teV system 110 is software installed on one or 
more servers. It is conceptually much lik\ application under test 114. In a preferred 
embodiment, test system 1 10 is a JAVA appli^tion. Like application under test 1 14, test 
system 1 10 is controlled through a graphical us\r interface 150. GUI 150 might be a web 
server as known in the art. One or more applica^n developers or test engineers might 
access test system over the network 122. In FIG. l\PC's 152 and 154 are PC's used by 
testers who will control the test process. 

Like application under test 114, multiple indivi(kals might use test system 110 
simultaneously. For example, multiple testers might be testing a single application. Each 
tester might be focused on testing different aspects of the application. Alternatively, each 
tester might be testing a different application. Numerous applications might be installed 
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on cUputers on network 122. Test system 110 might be testing different applications at 
the same time. 

TOming now to FIG. 2, details of test system 110 are shown. Test system 110 
performs s^eral functions. One function is the generation of test code. A second 
function is t\ execute the test code to exercise one or more EJB component's in the 
application undi^ test. Another fimction is to record and analyze the results of executing 
the test code. These functions are performed by software running on one or more 
computers connect^ to network 122. The software is written using a commercially 
available language to\erform the functions described herein. 

FIG. 2 shows th^^test system 110 has a distributed architecture. Software 
components are installed oK several different computers. Multiple computers are used 
both to provide capability forWultiple users, to a allow a user to perform multiple tasks 
and also to run very large tests.Vie specific number of computers and the distribution of 
software components of the test\?ystem on those computers is not important to the 
invention. \ 

Coordinator 210 is a software ap^cation that interfaces with GUI 150. The main 
purpose of coordinator 210 is to route useArequests to an appropriate server in a fashion 
that is transparent to a user. Turning to pb. 3, coordinator 210 is shown in greater 
detail. It should be appreciated, though, that Vg. 3 shows the conceptual structure of 
coordinator 210. Coordinator 210 might not beSa single, separately identified piece of 
software. It might, for example, be implemented as coordination software within the 
various other components of test system 110. Alk it should be realized that a web 
server used to implement GUI 150 also provides coorLation functions, such as queuing 
multiple requests from an individual or coordinating multiple users. 

Coordinator 210 contains distribution unit 312. Distidbution unit 312 is preferably 
a software program running on a server. As user requests W received from GUI 150, 
they are received by distribution unit 312. As the requests ark received, distribution unit 
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312 determines the type of resource needed to process the request. For example, a 
request to^nerate code must be sent to a server that is running a code generator. 

Coordbator 210 includes several queues to hold the pending requests. Each 
queue is implemented in the memory of the server implementing coordinator 210. In 
FIG. 3, queues 3yi8A...318C are illustrated. Each queue 318A...318C corresponds to a 
particular type oXresource. For example, queue 318A could contain code generator 
requests, queue 318^ could contain test engine requests and queue 318C could contain 
data analysis request^ Distribution unit sends each request to one of the queues 
3 1 8A. . .3 1 8C, based on the type of resources needed to process the request. 

Associated with eaclKflueue 318A...318C is queue manager 320A...320C. Each 
queue manager is preferabW implemented as software running on the server 
implementing coordinator 21oV)r the server implementing the relevant piece of 
coordinator 210. Each queue manager maintains a list of servers within test system 1 10 
that can respond to the requests inNls associated queue. A queue manager sends the 
request at the top of the queue to a s\ver that is equipped to handle the request. The 
connection between the queue manager aKd the servers equipped to handle the requests is 
over network 122. If there are other serWs available and still more requests in the 
queue, the queue manager will send the next\equest in the queue to an available server. 
When there are no available servers, each queu\^anager waits for one of the servers to 
complete the processing of its assigned request. \ 

As the requests are processed, the servers, sucH^ the code generators and the test 
engines report back to the queue managers. In respbnse, the queue managers send 
another request from the queue and also provide the resiL back to the distribution unit 
312. Distribution unit 312 can then reply back to the\ser that issued the request, 
indicating that the request was completed and either givinW the results or giving the 
location of the results. For example, after test code is generate, the user might receive 
an indication of where the test code is stored. After a test is ^ecuted, the user might 
receive a report of the average execution time for the test or the Idtation of a file storing 
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ach m^surement made dviring the test. 

It will be appreciated by one of skill in the art that software systems that process 
user comma^s, including commands from multiple users, are well known. Such 
systems must have an interface for receiving commands from a user, processing those 
commands and\esenting results to the user. Such interfaces also allow those results to 
be used by the usdr for implementing fiirther commands. Such an interface is employed 
here as well and isVpicted generally as GUI 150. For example, GUI 150 will allow a 
user to enter a comiWd that indicates code should be generated to test a particular 
application. Once the\ode is generated, GUI 150 allows the user to specify that a test 
should be run using that tfest code. 

It is possible that sorte requests will require the coordination of multiple hardware 
elements. As will be descried hereafter, one of the fimctions of the test engines is to 
apply a load that simulates multiple users. In some instances, one computer can simulate 
multiple users by running multip^client threads. However, there is a limit to the number 
of client threads that can run on a seVer. 

FIG. 4 shows the process by wHich queue manager 320B coordinates the actions 
of test engines located on separate serveV At step 410, queue manager 320B waits for 
the required number of test engines to bVome available. Once the test engines are 
available, at step 412 queue manager 320B sdnds commands to each test engine that will 
be involved in the test to download the test cdde from the appropriate one of the code 
generators 212A and 212B. \ 

Queue manager 320B then begins the process of synchronizing the test engines 
located on different servers. Various methods could Be used to synchronize the servers. 
For example, if very great accuracy is required, each se\er could be equipped with radio 
receivers that receive satellite transmissions that are intVided to act as time reference 
signals, such as in a GPS system. Calibration processes Wld alternatively be used to 
determine the amount of time it takes for a command to reaV and be processed by each 
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server. Commands could then be sent to each server at times offset to make up for the 
time differences. In the preferred embodiment, a simple process is used. At step 414, 
queue n^ager sends a message to each server to be acting as a test engine. The message 
asks that server to report the time as kept by its own internal circuitry. 

Upon\eceiving the internal time as kept by each of the servers, at step 416 queue 
manager 320B\dds the same offset to each local time. At step 418, queue manager 418 
sends the offset titaes back to the servers. The offset local times become the local starting 
time of the test fo^each server. Each server is instructed to start the test when its local 
time matches the dffset local time. In this way, all the servers start the test 
simultaneously. \ 

Queue manager 320B waits for the execution of the test code at block 420. Some 
test cases will entail multipleSests. At step 422, a check is made of whether the particular 
test case being executed requir^more tests. For example, as will be described in greater 
detail below, one kind of test c^that test system 1 10 will run is a load test. During a 
load test, multiple test engines,\ach executing multiple client threads, execute to 
simulate multiple users accessing application under test 1 14. An operating parameter of 
application under test 1 14 is then measVd. In the preferred embodiment, the number of 
simultaneous users being simulated can ^varied and the operating parameter measured 
again. Taking data at multiple load conditions allows test system 110 to determine the 
affect of load on application under test 1 14. Measurements of these types require that a 
full test case include multiple repetitions of the sWe test with different load conditions. 

If there are more conditions under whichUhe test should be run, execution 
proceeds to step 424. At step 424, the number of clierit threads is increased as needed for 
the new test case. Execution then loops back to step 4V At step 410, queue manager 
320B waits for the required number of test engines to be aWlable. When the hardware is 
available, the test is performed through steps 412, 414, 4^, 418 and 420 in the same 
manner as for the first test condition. At step 422, a checl^is for whether the request 
entails multiple test conditions. If there are no further test coMtions, the request fi-om 
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queue\l8B is considered complete. If there are further test conditions that need to be 
run to complete the request, the process is again repeated. 

For t\t system 1 10 to operate, it is necessary that there be test code. A user could 
provide test coke. Or, test code could be provided by automatic code generation systems, 
such as TESTIvMSTER sold by Teradyne Software and System Test of Nashua, NH. 
However, FIG. 2 mustrates that code generators 212A and 212B are used in the preferred 
embodiment to create the code. Turning to FIG. 5, greater details of a code generator 212 
are shown. \ 

Code generator Til contains several scripts 512. Each script is a sequence of 
steps that code generator 212 must perform to create code that performs a certain type of 
test. The scripts can be pr^ared in any convenient programming language. For each 
type of test that test system irO will perform, a script is provided. User input on the type 
of test that is desired specifies Viich script 512 is to be used for generating code at any 
given time. \ 

The selected script 512 assembles test code 516. The information needed to 
assemble test code 516 comes from sevVal sources. One source of information is the test 
templates 514. There are some steps thfat are needed in almost any kind of test. For 
example, the object being tested must beVployed and some initialization sequence is 
required. If the tests are timed, there must bk code that starts the test at a specified start 
time and an ending time of the test must beVcorded. Also, there must be code that 
causes the required data to be logged during theVst. After the test, there might also be 
some termination steps that are required. For exVple, where the initialization started 
with a request for a reference to a particular EJB Wonent, the test code will likely 
terminate with that reference being released. The te^code to cause these steps to be 
performed is captured in the set of templates 514. \ 

In addition, there might be different templates to dhsure that the test code 516 
appropriately reflects inputs provided by the user. For exWple, different containers 
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Vilnight re\ire different command formats to achieve the same result. One way these 
(different foWs can be reflected in the test code 516 is by having different templates for 
\ach contained Alternatively, a user might be able to specify the type of information that 
is to be recorded during a test. In that instance, a data logging preference might be 
implemented by Having a set of templates that differ in the command lines that cause data 
to be recorded during a test. 

The templatesVe written so that certain spaces can be filled m to customize the 
code for the specific oBiect to be tested. In the preferred embodiment, code generator 
212 generates code to te^a specific EJB component in an application under test. One 
piece of information that Will need to be filled in for many templates is a description of 
the EJB component being tekd. Another piece of information that might be included is 
user code to put the applicationWder test in the appropriate state for a test. For example, 
in testing a component of an application that manages a database of account information 
for a bank, it might be necessary tdyhave a specific account created in the database to use 
for test purposes or it might other^se be necessary to initialize an application before 
testing it. The code needed to cause these events might be unique to the application and 
will therefore be best inserted into the t\st code by the tester testing the application. In 
the illustrated embodiment, tiiis code is\serted into tiie template and is then carried 
through to the final test code. 

The template might also contain spaces for a human tester to fill in other 
information, such as specific data sets to use f^a test. However, in the presently 
preferred embodiment, data sets are provided by th^human user in the form of a data 
table. 

Code generator 212 could generate functional tesV Functional tests are those 
tests that are directed at determining whether the bean corispctly performs its required 
functions. In a fimctional test, the software under test is exerciW with many test cases to 
ensure that it operates correctiy in every state. Data tables indicating expected outputs 
for various inputs are used to create fimctional test software. HoVever, in the presentiy 
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Dreferfed embodiment, code generator 212 primarily generates test code that performs 
load tesk In a load test, it is not necessary to stimulate the software under test to 
exercise eVtery possible function and combination of fimctions the software is intended to 
perform. IlWer, it is usually sufficient to provide one test condition. The objective of 
the load test ik to measure how operation of the software degrades as the number of 
simultaneous usas of the application increases. 

In the preferised embodiment, test system 110 contains scripts 512 to implement 
various types of loadVsts. One type of load test determines response time of an EJB 
component. This allovb the test system to vary the load on the EJB component and 
determine degradation ofVesponse time in response to increased load. Another type of 
load test is a regression typVoad test. In a regression type test, the script runs operations 
to determine whether the EJB component responds the same way as it did to some 
baseline stimulus, hi general, tfke response to the baseline stimulus represents the correct 
operation of the EJB componentX Having a regression type test allows the test system 
1 10 to increase the load on a bean aiW determine the error rate as a fimction of load. 

To generate test code 516 for th^e types of load tests, the script 512 must create 
test code that is specific to the bean unde^test. The user provides information on which 
bean to test through GUI 150. In the prefeirM embodiment, this information is provided 
by the human tester providing the name of thdSfile within the application under test that 
contains the "deployment descriptor" for the spWic bean under test. This information 
specifies where in the network to find the beaia under test. Script 512 uses this 
information to ascertain what test code must be generated to test the bean. 

Script 512 can generate code by using the attribiJtes of the platform independent 
language in which the bean is written. For the example \f Sun JAVA language being 
used here, each bean has an application program interface Jelled a "reflection." More 
particularly, each bean has a "home" interface and a "remotV interface. The "home" 
interface reveals information about the methods for creating or ffi^ing a remote interface 
in the bean. The remote interface reveals how this code can bk accessed from client 
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^/-^ftware. Of Vticular interest in the preferred embodiment, the home and remote 
interfaces providd\the information needed to create a test program to access the bean. 

Using the reflection, any program can determine what are known as the 
"properties" and "methods" of a bean. The properties of a bean describe the data types 
and attributes for a variable used in the bean. Every variable used in the bean must have 
a property associated wifh it. In this way, script 512 can automatically determine what 
methods need to be exerciki to test a bean and the variables that need to be generated in 
order to provide stimulus toWie methods. The variables that will be by the methods as 
they are tested can also be detained. In the preferred embodiment, this information is 
stored in symbol table 515. 

Symbol table 515 is a file inVy convenient file format. Many formats for storing 
tabular data are known, such as .xml format. Once the information on the methods and 
properties are captured in a table, script^fi 12 can use this information to create test code 
that exercises the methods and propertie\ of the particular component under test. In 
particular, script 512 can automatically create a variable of the correct data type and 
assign it a value consistent with that type for aW variable used in the bean. 

FIG. 5 shows a data generator 518. D^a generator 518 uses the information 
derived fi-om the reflection interface to generate vkes for variables used during testing 
of a bean. There are many ways that appropriate Vlues could be generated for each 
variable used in the test of a particular bean. HoweverV the commercial embodiment of 
the present invention, the user is given a choice of thrV different algorithms that data 
generator 518 will use to generate data values. The \ser can specify "maximum," 
"minimum" or "random." If the maximum choice is sWied, data generator 518 
analyzes the property description obtained through the reflect^ interface and determines 
the maximum permissible value. If the user specifies "mininUn" then data generator 
518 generates the smallest value possible. If the user specifiesWdom, data generator 
5 1 8 selects a value at random between the maximum and the minim\ 
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In \ny instances where a load test is desired, the exact value of a particular 
variable is ndt important. For example, when testing whether a bean can properly store 
and retrieve a\alue from a database, it usually does not matter what value is stored and 
retrieved. It onl\ matters that the value that is read from the database is the same one that 
I was stored. Or, Xhen timing the operation of a particular bean, it will often not matter 
\what values are Lut to the method. In these scenarios, data generator 518 can 
automatically generate the values for variables used in the test code. 

In cases where tke specific values of the variables used in a test are important, 
code generator 212 provides the user with another option. Rather than derive values of 
variables from data generat^518, script 512 can be instructed to derive data values from 
a user provided data table 52^ A user might, for example, want to provide a data table 
even for a load test when the execution time of a particular function would depend on the 
value of the input data. 

A data table is implemented dimply as a file on one of the computers on network 
122. The entries in the table, specifyVg values for particular variables to use as inputs 
and outputs to particular methods, are\eparated by delimiters in the file. A standard 
format for such a table is "comma s^arated values" or CSV. In a preferred 
embodiment, test system 110 includes a file editor - of the type using conventional 
technology - for creating and editing suchVile. In addition, test system 110 would 
likely include the ability to import a file - aga^ using known techniques - that has the 
required format. 

The methods of a bean describe the functionsSlhat bean can perform. Part of the 
description of the method is the properties of the variables that are inputs or outputs to the 
method. A second part of the description of each method^^ which can also be determined 
through the reflection interface - is the command needed t^voke this method. Because 
script 512 can determine the code needed to invoke any metVd and, as described above, 
can generate data values suitable to provide as inputs to tl^t method, script 512 can 
generate code to call any method in the bean. 
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In tWe preferred embodiment, directed at load testing, the order in which the 
methods of aVean are called is not critical to an effective test. Thus, script 512 can 
automatically generate useful test code by invoking each method of the bean. The order 
in which the meAods are invoked does not matter if the only parameter that is measured 
is the time it takes the methods to execute. 

More sophistidfated tests can be automatically built by relying on the prescribed 
pattern for the languagA In Sun JAVA, entity beans for controlling access to a database 
should have methods thaUave a prefix "set" or "get". These prefixes signal that the 
method is either to write dkta into a database or to read data from the database. The 
suffix of the method name indicates which value is to be written or read in the database. 
For example, a method nameck setSSN should perform the function of writing into a 
database a value for a parameter Mentified as SSN. A method named getSSN should read 
the value from the parameter name(i SSN. 

By taking advantage of these ^escribed patterns, script 512 can generate code to 
exercise and verify operation of both Whods. A piece of test code generated to test 
these methods would first exercise theViethod setSSN by providing it an argument 
created by data generator 518. Then, the rilethod getSSN might be exercised. If the get 
method returns the same value as the argument that was supplied to the set method, then 
it can be ascertained that the database access executed as expected. 

For many types of enterprise wide appl«>ations, the beans most likely to be 
sensitive to load are those that access the databak Thus, testing only set and get 
methods provides very useful load test information. \ 

However, the amount of testing done can be exp^ded where required. Some 
beans also contain methods that create or find rows in Liatabase. By convention, 
methods that create or find rows in a database are named startW with "create" or "find." 
Thus, by reflecting the interface of the bean, script 512 can ako determine how to test 
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these riiethods. These methods can be exercised similarly to the set and get methods. 
The properties revealed through the application interface will described the format of 
each row i\the database. Thus, when a create method is used, data can be automatically 
generated to fill that row, thereby fully exercising the create method. 

In a preftirred embodiment, find methods are exercised using data from a user 
supplied data tablA520. Often, databases have test rows inserted in them specifically for 
testing. Such a testVw would likely be written into data table 520. However, it would 
also be possible to creV a row, fill it with data and then exercise a find method to locate 
that row. \ 

Once the commandsVat exercise the methods of an EJB component are created, 
script 512 can also insert into tke client test code 516 the commands that are necessary to 
record the outputs of the test. If a test is checking for numbers of errors, then test code 
516 needs to contain instructionsVat record errors in log 216. Likewise, if a test is 
measuring response time, the test cdde 516 must contain instructions that write into log 
216 the information fi:om which response time can be determined. 

In the described embodiment, alWajor database fiinctions can be exercised witii 
no user supplied test code. In some insUces, it might be possible to exercise all the 
fimctions with all test data automatically ge\^rated. All the required information could 
be generated from just the object code of the aVication under test. An important feature 
of the preferred embodiment is that it is "minim^ invasive" - meaning that very little is 
required of the user in order to conduct a test and ti^ test does not impact tiie customer's 
environment. There is no invasive test harness. The diient code runs exactly like tiie code 
a user would write. \ 

In some scenarios, it will be necessary or desirable for a user to insert specific 
steps into the test code 516. Test system 110 allows for ^ user to edit test code 516. 
For example, some beans will perform calculations or peVorm functions other than 
database access. In tiiese instances, the user might chose to insVt test code tiiat exercises 
these fiinctions. However, because bottlenecks in the opeAtion of many N-tiered 
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applications dccur in the entity beans that manage database transactions, the simple 
chniques descldbed herein are very useful. 

Once test cdde 516 is generated, with or without editing by a user, test system 110 
is ready to execute \est. User input to coordinator 210 triggers the test. As described 
above, coordinator 2l\ queues up requests for tests and tests are executed according to 
the process pictured in W. 4. To simulate one user, the test code 516 is executed as a 
single thread on one of\he test engines 214A...214C. To simulate multiple users, 
multiple threads are initiatedon one or more of the test engines 214A.. .214C. 

The results of any testsVe stored in log 216. FIG. 2 shows log 216 as a separate 
hardware element attached to rietwork 122. Log 216 could be any type of storage 
traditionally found in a network,\uch as a tape or disk drive attached to a computer 
server. For simplicity, it is shown a^a separate unit, but could easily be part of any other 
server in the network. 

Many types of data could be stoAd in log 216. For example, there are several 
possible ways that "response time" could b\ measured. One way is that the total time to 
execute all the methods in a bean could be Measured. Another way is that the start up 
time of a bean could be measured. The startuto time is the time it takes from when the 
bean is first accessed until the first method is ak to execute. Another way to measure 
response time is to measure the time it takes forVach method to execute. As another 
variation, response time could be measured based\m how long it takes just the "get-" 
methods to execute or just the "set-" methods to execiJte. 

Different measurements must be recorded, deluding on which measure of 
response time is used. For example, if only the total r^ponse time is required, it is 
sufficient if the test code simply records the time that the Vrtion of the test code that 
exercises all the methods starts and stops executing. If th\ startup response time is 
required, then the client test code must record the time that it firWcesses the bean under 
test and the time when the first method in the test sequence is re^y to be called. On the 
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other haiid, if the response time is going to be computed for each method, the client test 
code musVecord the time before and after it calls each method and some indication of 
the method\eing called must also be logged. Similar information must be recorded if 
responses ofVst "get-" or "set-" fiinctions are to be measured, though the information 
needs to be recdrded for only a subset of the methods in these cases. 



V 



In additioiAwhen there are multiple users being simulated, there are multiple 
values for each data\oint. For example, if test system 110 is simulating 100 users, the 
time that it takes for\he bean to respond to each simulated user could be different, 
leading to up to 100 diffei^ent measurements of response time. The response time for 100 
users could be presented asS^e maximum response time, i.e. the time it takes for all 100 
simulated users to finish exeXising the bean under test. Alternative, the average time to 
complete could be reported as\ie response time. As another variation, the range of 
values could be reported. 

In the preferred embodiment, client test code 516 contains the instructions that 
record all of the information that wouldW needed for any possible measure of response 
time and every possible display formatN. The time is recorded before and after the 
execution of every method. Also, an indication that allows the method to be identified is 
recorded in log 216. To support analysis base^n factors other than delay, the actual and 
expected results of the execution of each metftod are recorded so that errors can be 
detected. Also, the occurrences of exceptions areSalso recorded in the log. Then, data 
analyzer 218 can review the log and display the res^nse time according to any format 
and using any definition of response time desired. O^the data analyzer can count the 
number of exceptions or errors. 

Once the data is stored, the user can specify the deslted format in which the data 
is to be presented. Data analyzer 218 accepts commands froV the tester concerning the 
format of the output and analyzes the data appropriately. In a pWerred embodiment, test 
system 110 has the ability to present the results of tests graphi^y ^ aid the tester in 
understanding the operations - particularly performance bottleneckV of appHcation under 
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Data analyzer 218 generates output in several useful formats. One important 
5|utput \s\a response time versus load graph. Log file 216 contains the starting and 
stoppingNiriies of execution tests for a particular test case. The test case includes the 
same measuLnents at several different load conditions (i.e. with the test engines 
214A...214C sftnulating different numbers of simultaneous users). Thus, data analyzer 
can read through\the data in log 216 and identify results obtained at different load 
conditions. This dala can be graphed. 

Another useful Valysis is the number of errors per second that are generated as a 
function of the number o^ simultaneous users. To perform this analysis, test code 516 
could contain instructionsVat write an error message into log 216 whenever a test 
statement produces an incorrdrt result. In the database context, incorrect results could be 
identified when the "get" fimction does not return the same value as was passed as an 
argument to the "set" function.Xor, errors might be identified when a bean, when 
accessed, does not respond or respWids with an exception condition. As above, data 
analyzer 218 can pass through the Idg file, reading the numbers of errors at different 
simulated load conditions. If desired, th^errors can be expressed as an error count, or as 
an error rate by dividing the error count bjNhe time it took for the test to run. 

Some examples of the types of out^s that might be provided are graphs 
showing: transactions per second versus numbik of users; response time versus number 
of users; exceptions versus numbers of users; er\rs versus numbers of users; response 
time by method; response time versus run time a\tramactiom per second versus run 
time. Different ways to measure response time were\Mscussed above. In the preferred 
embodiment, a transaction is defined as the execut\of one method though other 
definitions are possible. \ 

Run time is defined as the total elapsed time in runnir^gthe test case, and would 
include the time to set up the execution ofEJB components. Viewing response time as a 
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function ofelapsMtime is useful, for example, in revealing problems such as "memory 
leaks". A memoXleak refers to a condition in which portions of the memory on the 
server running the duplication under test gets used for unproductive things. As more 
memory is used uXoductively. there is less memory available for running the 
A application under test \d execution slows over time. Alternatively, viewing results in 
^ this format might reveal that the application under test is effectively utilizing caching. If 
^ \ caching is used effectively, f^e execution time might decrease as elapsed time increases. 

Turning now to FIG. 6\peration of test system 1 10 from the tester's perspective 
is illustrated. FIG.6 iUustratesVthe tester interface to be a traditional web browser 
interface as it would appear on thXerminal of PC 152 or 154. This type of interface is 
the result of using a web server to\implement GUI 150 with commercially available 
browser software on the tester PC's isiand 154. 

Element 612 is the browser toolWu". The browser provides the ability for the 
tester to perform such functions as printingVd connecting to pages presented by various 
servers in the system. The tester performs th^e functions through the use of the tool bar 
612. Field 614 indicates the server to which thW 152 or 154 is currently connected. In 
the illustrated example, field 614 contains the\ddress on network 122 for the server 
containing GUI 150. \ 

Window 610 is the area of the screen of V: 152 or 154 where information 
specific to test system 110 is displayed. As with tradifl^nal web based applications, this 
information is displayed as the result of HTML files thWe downloaded over network 
122. Certain elements displayed in window 610 repr^ent hyperlinks, which when 
accessed by a tester cause other pages to be downloaded s^at different information is 
displayed for the tester. Other elements displayed in windoV 610 represent data entry 
fields. When a human tester fills in these fields, information iksubmitted to test system 
110. \ 



Tabs 616 represent choices a tester can select for operating ri^de. FIG. 6 




three tabs. There i\one tab for "setup", one for "test case" and one for "results". Each of 
the tabs is hyperlinked. These hyperlinks connect to pages on servers in the network that 
are programmed to nbnage the test system through a particular phase. Setup is for 
configuring the test system, such as by identifying addresses for servers on network 122 
that contain test enginesA "Test case" is for creating and running a particular test case, 
which in the preferred embodiment means test code for a particular bean and parameters 
specifying how that test codfe is to run. "Results" is for viewing the results of a test case 
that has executed. In FIcV, the "SETUP" tab is shown selected, meaning that the 
balance of window 610 displ^s hyperlinks and fields that are appropriate for entering 
data or commands for the SETUP phase. 

Region 618 contains a s^of hyperlinks. These hyperlinks present a list of 
choices to the user about what caA be setup. Selecting one of these hyperlinks will 
change the information appearing in region 620. It is well known in the art of web based 
software to have hyperlinks on one pkrt of window change information appearing in 
another part of the window. 

In the Example of FIG. 6, the hAerlink "Machine" in region 618 has been 
selected. Therefore, region 620 contains fields that can be used to identify a machine to 
be added to test system 110. In FIG. 6, a screeh to add a client host computer is shown. 
A client host computer acts as a test engine 214AV .214C. 

Region 618 also contains hyperlinks for 'Vrojects" and "Data Tables". As 
described above, test system 110 can be used by miJUiple testers or can be used to test 
multiple applications under test. To segregate the info)ttiation relating to various tasks, a 
"Project" is defined. All information relating to a particidar project is identified with the 
project. In this way, test system 110 can identify infonAation that is logically related. 
For example, test code developed to test a particular bean iiWi application will be given 
the same project name as test code to test other beans in th^application. In this way, 
general information about the application - such as the path to\he server through which 
the application can be accessed is stored once with the project ratlW than with each piece 




of test code\ As another example, the test results can be associated with the test code that 
generated them through the use of projects. 

The hyperUnk for "Data Tables" allows the tester to identify to the system where 
data tables are stofed to test specific beans. In general, the tester will create the data 
tables by hand or automatically apart from test system 110 based on the features that the 
tester desires to haveVsted. The data tables will be stored in files on a computer on 
network 122. Before u^ by test system 1 10, the tester must provide the network address 
for that file. 

Field 622 is a menu field.'.slt is a drop down^menu box. When accessed, it will 
display a menu of choices of tfee actions the tester can specify for test system 110. The 
contents of the menu is context sWsitive, meaning that for any given page, only the menu 
choices that are appropriate are diVayed. Actions that a user might want to choose from 
the menu are things such as edit, delW, add new, rename, etc. 

Turning now to FIG. 7, a screeAof tester PC 152 or 154 is shown when the TEST 
CASE tab selected. Selecting the TEStVaSE tab allows the tester to specify what is to 
be tested and how the test is to be conducted. In the example of FIG. 7, window 610 
contains information that describes a test c^e. This particular page is displayed when the 
"edit" has been selected in menu field 622. Xpield 710 indicates the name of the project 
to which the test case is associated. Field IW is a screen display element known as a 
drop down box. When activated by a tester, f)bld 710 will become a list of all projects 
that have been previously created by and tester u^jng test system 110. As shown in FIG. 
7, a project called "Demo Project" is being used. 

Field 712 identifies the particular test case bV name. Field 712 is also a drop 
down box, allowing previously defined test cases to bd^elected from a list. In addition, 
one of the options that appears when the drop down \ox is accessed is "<New Test 
Case>". When selected, a new test case is created and information about this test case 
can be entered. This type of menu is common for programs ^th graphical user interfaces 
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and is used at\ 



jeveral points in the interface for presenting choices to a human tester. 



In the ekample of FIG. 7, a test case "Customer Test" has been created and 
i^ormation has already been entered. 

In region there are a series of fields in which data can be entered or changed 
to define or modify the test case. In field 716, a name can be provided for the test case. 
This name allows the test case to be referenced in other parts of the test system. 



In field 718, a destription of the test case can be entered. This description is not 
used by the automatic features of the test system, but can act as a note to a human tester 
to signify what the test case dbes or why it was created. Field 720 is hkewise not used by 
the automated system. HowevV, this field holds the name of an individual who authored 
the test case to facilitate other acHninistrative functions. 

Field 722 is a "radio button\type field. It presents a tester with a list of choices, 
only one of which can be selected at V time. In this example, field 722 allows the tester 
to specify the type of test. As previdusly described, code generators 21 2 A and 21 2B 
contain a plurality of scripts 512 that geiierate test code. As described above, the script 
assembles templates and generate commaAd lines for a particular type of test that is to be 
conducted. Thus, the tester must specify t\e test type in order to allow the appropriate 
script to be selected. In this example, only t>^ types of tests are presented as options to a 
tester - a load test and a fimctional test. Thesktypes of tests were discussed above. Field 
724 allows the tester to specify a "deployment dtescriptor." In the JAVA language, every 
bean has associated with it a deployment descriptor. The deployment descriptor is a file 
that identifies the bean and defmes the parameters ^th which the EJB component will be 
deployed on the applications server. Examples of\the parameters are tiie number of 
instantiations of the EJB component with which\o start the applications server 
(sometimes called a "pooling number") and how much rt^emory is allocated to the bean. 
These functions are performed by the container 130. 



The tester provides tiie deployment descriptor by entefdng the patii to a file on 
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network m. The test system 1 10 reads the deployment descriptor to find the name 
the bean under test and then to access the bean through reflection to determine 
Tnethods and properties. 



^ Field 726 ^lows the tester to specify the type of data to be used in creating the 
test code 516. a\ described above, in the preferred embodiment, the data can be 
automatically generated by test system 1 10 or can be supplied through a data table. For 
automatic data generation, the data can be generated by using the maximum possible 
value of each variable, tHe minimum possible value or a value randomly selected between 
the maximum and the minimum. Alternatively, the data can be specified by a data table. 
In FIG. 7, field 726 indicated that tester desires test system 1 10 to generate data using the 
data table named dd.csv. l\ the tester had wanted the test system to automatically 
generate data, the tester would Nspecify in field 726 whether the data was to be generated 
randomly or whether the maximtnn or minimum values were to be used. 

Field 728 allows the user to Soecify the server on which the application under test 
runs. FIG. 1 shows that the beans isiof the application under test are within a container 
130. In this context, "server" refers to the software that creates the container for the 
application. For each platform indepeMent language, there are a limited number of 
commercially available servers. Test sysffem 110 contains script files that will generate 
the appropriate test code for any server. Wmie most of the client test code will be server 
independent, it is possible that servers will implement certain fimctions in unique ways. 
Thus, there needs to be script files that accoiJ^ for the fimctions that are implemented 
differently on certain servers. \ 

FIG. 8 shows a screen display when the tester has used menu field 622 and 
selected to have the deployment descriptor for a test c^e to be displayed. If desired, the 
tester can then edit the deployment descriptor to try altertiative configurations. 

FIG. 9 shows a screen display when a tester has selected fi-om menu field 622 to 
have the generated test code displayed. In FIG. 9, the teWode 516 is identified as 




30 




"client code" bfecause it will simulate the operation of a client 120 of the application 
under test 1 14. Ihe displayed code corresponds to the code generated for the project and 
tast case identified in fields 710 and 712. In this screen, the tester can also edit the test 
fode. 

One instanceVvhen a tester might desire to edit test code is when most of the 
commands in the testVode can be automatically generated, but certain commands must 
have specific data valuk for the application under test to fiinction properly. The tester 
could have test system WO automatically generate the test code, but then manually edit 
the few data values that mVtter. As an example, a particular bean might include a method 
that processes positive an\ negative values differently. The tester might know that 
processing negative number^takes longer and therefore change a randomly generated 
value to a negative number. 

An altemative scenario in Which a tester might wish to edit test code 516 is when 
the bean being tested contains methVls to be tested other than those that follow a pattern, 
such as the "set", "get", "create" an(k"find" methods described above. The tester might 
create test code that tests methods thatVo not follow the pattern. This code could then be 
inserted by the human tester into the gentoated test code. 

FIG. 10 shows a screen display for another function performed by a human tester 
while specifying the TEST CASE, also sele\ted through menu field 622. FIG. 10 is a 
screen display used when the test case is to beWun. The project and specific test case is 
specified by entries in fields 710 and 712. Info^ation about how to run the test case is 
entered in region 1010. 

Region 1010 contains several fields. Field 10^2 indicates the name of the file in 
log 216 where the results of executing the test case Will be stored. In this way, data 
analyzer 218 will be able to locate the data for a particular test case to analyze and 
present to a tester in the desired form. 
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Field M)14 gives the URL - or network address - for the appUcation under test. 
This informatidti could be identified by using the name for a particular machine that was 
.{jfSliously set-un by the human tester. 

\ Field 1016Vives the URL - or network address - for a server to use as a test 
engine. Again, the server could be identified by using the name for a particular machine 
that was previously sV-up by the human tester. The screen displayed in FIG. 10 is used 
for an embodiment of me invention where all test simultaneously executing copies of the 
client test code are runW a single machine. If test system 110 includes multiple test 
engines and automatical^ schedules execution of test code as described in conjunction 
with FIG. 4 above, then fieM 1016 is not required. 

Field 1018 gives the iriaximum number of concurrent users to be simulated at one 
time during the execution of th\ test case. Field 1020 allows the user to specify the rate at 
which the number of simultaneous users will be simulated during a test. In the example of 
FIG. 10, the test case will be\ completed after 100 users have been simulated 
simultaneously and the number of simultaneous users will increase by 10 each time the 
test code is run. For the examples gfven here, 10 copies of the client test code shown in 
FIG. 9 will first be executed simultanfeously. Then 20 copies of that test code will be 
executed simultaneously. The test code Vill be repeated for 30, 40, 50, 60, 70, 80, 90 and 
100 simultaneous users before the test cassis completed. 

After the test case has been run, the t^ter can view and analyze the results. The 
human tester can access the fvinctions of the test system 1 10 that display and analyze 
results by selecting the RESULTS tab from tabs\616. FIG. 11 shows a page displayed 
when the RESULTS tab is selected. The page sho^ in FIG. 1 1 is for when the tester has 
requested to see summary data through use of menu neld 622. 

Fields 710 and 712 are also present on the We shown in FIG. 11. This 
information is used by the test system to locate the a^ropriate data to display. In 
addition, there is a field 1 1 10 that allows the user to enter the name of the results file to 
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display. As Wscribed in conjunction with FIG. 10, the user specifies in field 1012 the 
name of a resmts file to hold the results of a particular run of a test case. The name of the 



ts file for cJesired run of the test is entered in filed 1110. 



\ 



FIG. 11 sHbws that window 610 contains a region 1112 that lists the results of a 
run of a particular t\st case in summary fashion. Part of the information in region 1 1 12 is 
simply a display of information input by the tester when editing or running the test case. 
For example, the target container, or the container 130 of applicafion under test 114 is 
listed. The maximum number of concurrent users simulated during the test is also 
displayed. The file cont^ning the test data use for the run of the test case that generated 
the resuhs is also displayeci as is the deployment descriptor. These values are displayed 
for ease of use by the humamtester. 

Region 1112 also includes information that was developed by data analyzer 218 
fi-om the data gathered for the specified run of the test case. In FIG. 11, the pieces of 
information that are displayed are the average response time and the maximum response 
time. As described above, as a test Case runs, the start and stop times of the execution of 
the test code is recorded in log 216. When the test code is run multiple times, each time 
simulating a different number of usertk the start and stop time is recorded for each 
number of users. Data analyzer can detWine the response time by simply computing 
the difference between the start and stop tinges. Once values are determined, they can be 
averaged or the maximum can be identified. 

FIG. 12 shows a different page that can b\ selected by the user to see the results in 
graphical format. As above, fields 710, 712 and iVlO allow the user to specify which run 
of a particular test case is used to create the results. \The graphical display in FIG. 12 is a 
graph showing numbers of transactions per second as the dependent variable with the 
number of simultaneous users as the independent vafiable. As described above, the 
information needed to compute the data values for this grMi is stored in log 216 after the 
test case is run and data analyzer 218 can retrieve it. For creation of the graph in FIG. 12, 
transactions per second was defined as the average numbeV of methods executed per 
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second per user. This value is essentially the reciprocal of response time. 

FIG.Vs shows a screen useful for displaying results to a human tester in a slightly 
different emb\diment. As with FIG. 12, the screen display in FIG. 12 is accessed when 
the "results:' tab is selected. Also like in FIG. 12, the page shown in FIG. 13 

^"'Ticludes fields 712 and 1 1 10 that allow the human tester to specify which results are 

'^ to be displayed. \ 

P \ \ 

The page showW in FIG. 13 includes an alternative way for the user to specify the 
format of the display. The screen in FIG. 13 includes menu fields 1310 and 1312. Menu 
field 1310 allows the tester to specify the manner in which response time is to be 
calculated. In FIG. 13, a vVe of "total" has been selected in field 1310. As described 
above, the "total" response tiriie is measured as the time firom first access of a bean until 
all methods of the bean have been exercised. Other choices in menu field 1310 allow a 
tester to specify that resuh be dkplayed for different measures of response time. As 
described above, the presently pretferred embodiment can measure response time just on 
the start up time or response time fOf individual methods or for get- functions and set- 
functions. \ 

Field 1312 allows a user to specm' the format of the display. In FIG. 13, the 
display is in HiLo format. In this format, results are displayed as bars, such as bar 1316. 
Each bar spans fi-om the fastest response tim\ to the slowest response time. A tick mark 
showing the average is also included in the illub-ation of FIG. 13. Other choices in menu 
field 1312 would, for example, allow the humanVester to see results in a line chart format 
as in FIG. 1 2 or in tabular format. \ 

Turning to FIG. 14, results in a tabular fonriat are shown. Field 1312 indicates 
that the display format of "Log File" has been selectk This format corresponds to the 
list shown in region 1412. The list contains a column At the name of the methods in the 
bean being tested. In this example, the data showA for each method reveals the 
minimum, maximum and average execution time for that n^thod. 
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As described above, test system 110 measures response time at various load 
conditions. The dispfeyed data represents response times at a particular load condition. 
^^hns, to make the list\he tester must specify the load condition for which data is to be 
displayed. To allow thib selection to be made, the page displayed in FIG. 14 contains a 
Vield 1410. In this field, a user can enter the load condition for which data is to be 
displayed. In this example, the human tester has entered a value of 500, indicating that 
500 threads executing the tfitst code were initiated in order to obtain the displayed data. 

Having described th A structure of test system 110 and giving examples of its 
application^ several important features of the test system 1 10 can be seen. One feature is 
that information about the peWormance of an application under test can be easily 
obtained, with much of the datdy being derived in an automated fashion. A software 
developer could use the test sykem to find particular beans that are likely to be 
performance bottlenecks in an appli\ation. The developer could then rewrite these beans 
or change their deployment descriptors. For example, one aspect of the deployment 
descriptor indicates the number of coMes of the bean that are to be instantiated within 
application under test 114. The develo^r could increase the number of instantiations of 
a bean if that bean is the bottleneck. 

The test system described herein prd^ides an easy and accurate tool to test EJB 
components for scalability. It creates a user specified number of virtual users that call the 
EJB component while it is deployed on the a^lications server. The tool does this by 
inspecting the EJB component under test andXautomatically generating a client test 
program, using either rules based data or data sullied by an a human tester, and then 
multithreading the client test program to drive the Effi component under test. The result 
is a series of graphs reporting on the performance ^rsus the number of users, which 
provide useful information in an easy to use format. 

Another feature of the invention is that the tests ar^run without requiring changes 
in the application under test or even the installation of speidal test agents on the server 
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ntaining the^ftware under test. The generated test code 516 exercises the bean in the 
plication undeXtest using remote procedure calls. 

Another feature of the described embodiment of test system 110 is that it is 
scalable. To increasAthe number of tests that could simultaneously be run or the size of 
the tests that could bd run, more test engines could be added. Likewise, more code 
generators could be adcted to support the simulation of a larger number of simultaneous 
users. The specific nufhber of copies of each component is not important to the 
invention. The actual nuiAber of each component in any given embodiment is likely to 
vary from installation to installation. The more users an application is intended to 
support, the more test enginesVe likely to be required. 

Another feature of the described embodiment is that testing is done on the 
simplest construct in the application under test - the beans in the illustrated example. 
There are two benefits to this approach. First, it allows tests to be generated very simply, 
with minimal human intervention. Second, it allows a software developer to focus in on 
the point of the software that needs\o be changed or adjusted in order to improve 
performance. 

It should be appreciated that displays of specific kinds of information have been 
described. Various other analyses might beWrformed. It was described that response 
times and error rates as a fimction of load could be graphed for display to a human tester 
for further analysis. One enhancement that miW be made to test system 1 10 is that the 
data analyzer 218 could be programmed to ^rform further analysis. It has been 
recognized that, as the load increases, tiiere is oftefi some point at which the performance 
of the system drastically changes. In some instanck the time to complete a transaction 
drastically increases. A drastic increase in transaction processing time indicates that the 
system was not able to effectively handle the load. However, a decrease in processing 
time can also indicate the load limit was reached. Sonietimes, a system under test will 
respond with an error message more quickly than it wduld take to generate a correct 
response. Thus, if the only parameter being tracked is \esponse time, a decrease in 
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V y^rocessing\ime as a function of load can also signal that the maximum load has been 
\7 [ exceeded. oV course, an increase in errors or error rate can also signal that the maximum 
load was exceeded. Data analyzer 218 could be used to identify automatically a 
maximum load for a particular test case. By running muhiple test cases, each test case 
focusing on a difftrent bean, test system 110 could automatically determine the bean that 
is the performance Bpttleneck and could also assign a load rating to application under test 
114. 

Having describeoV one embodiment, numerous alternative embodiments or 
variations might be madeS. For example, it was described that test system 110 
automatically generates test code to exercise beans that follow a partem for database 
access. These beans are sometimes called "entity beans." In general, there will be other 
beans in an application that perfoVn computations on the data or that control the timing 
of the execution of the entity beans\ These beans are sometimes called "session beans." 
Session beans are less likely to foUoW prescribed programming patterns that make the 
generation of test code for entity beansVmple. As a result, the automatically generated 
test code for session beans might no\ fully test those beans. In the described 
embodiment, it is expected that the humanSlester supply test code to test session beans 
where the automatically generated tests are inadequate. 

One possible modification to the described embodiment is that the completeness 
of tests for session beans might be increased. One way to increase the accuracy of tests 
for session beans would be to capture data about tl^ execution of those beans during 
actual operation of the application under test 114. THis data could allow an automated 
system to determine things like appropriate data values^ which might then be used to 
build a data table. Or, the captured data could allow the ^tomated system to determine 
the order in which a session bean accesses other session beai^s or entity beans to create a 
realistic test. 



Also, as described, test code is generated to test a particular bean, which is a 
simple construct or "component" of the application under test. Tha;. testing could focus 
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on different constructs, such as specific methods in a bean. Test code could be generated 
to test specific methods within beans. Or, it was described that the system records start 
and stop tin^ of the execution of the test code. The times of other events could be 
recorded instead or in addition. For example, start and stop times of individual methods 
might be recorded, allowing performance of individual methods to be determined. 

Altemativel\ the complexity of the constructs being tested could be increased. 
Multiple beans mighWje tested simultaneously to determine interactions between beans. 
For example, multiple test cases might be executed at the same time, with one test case 
exercising a specified instances of one bean and a different test case exercising a 
specified number of instancies of a second bean. 

As another example of \ possible variation, it was described that a human tester 
can insert code into a template t^o such things as put the application under test into a 
predictable state. Lines of code riught be inserted directly, for example by the user 
simply typing the lines of code. Or, me tester might insert a "tag" into the template. The 
tag would identify a code segment stor^ elsewhere within the test system. In this way, 
the same code segment could be included M multiple places in the template or in multiple 
templates. \ 

As another example of possible variations, the number of templates used to 
construct test code might be varied. One possibiW is that each template contains all of 
the steps needed to initialize, run and terminate a test. Thus, test code would be created 
by filling in a single template. Alternatively, each teVplate might contain only the steps 
needed to perform one function, such as initialization testing or termination, hi this 
implementation, test code would be created by stringing tWther multiple templates. 

Also, it was described that in running a test that a number of simultaneous users is 
"synchronized". Simultaneous users are simulated by synchtonizing copies of the test 
code on different servers and on the same server. The term "synWonized" should not be 
interpreted in so limited a way as to imply that multiple copi^ are each performing 
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exactly the same function at exactly the same time. Thus, when described herein that 
execution is sjcachronized, all that is required is that each copy of the code is making 
requests of the application under test during the window of time when the test is being 
executed. Some^opies of the code will likely start execution sooner or end sooner than 
the others. Howler, as long as there is overlap in the timing of execution, the test 
programs can be samto be synchronized or running concurrently. 

As a further vanation, it was described that the test system 1 10 provides outputs 
indicating the performan^ of an application under test as a function of load. These 
outputs in graphical or tabukr form can be used by an application developer to identify a 
number of concurrent users \t which problems with the application are likely to be 
encountered. Potential problerHs are manifested in various ways, such as by a sudden 
change in response time or error ra^ as a function of load. Test system 1 10 could readily 
be programmed to automatically idekify patterns in the output indicating these problem 
points. \ 

Another useful modification woulX allow test system 110 to aid in identifying 
settings for various parameters in the deploWent descriptor. As described above, the 
deployment descriptor for a bean identifies parameters such as memory usage and a 
"pooling number" indicating the number of instances of a bean that are created at the 
initialization of an application. These and other\ettings in the deployment descriptor 
might have an impact on the performance time andVmaximum load that an application 
could handle. One use of the test system described abo\e is that it allows a test case to be 
repeated for different settings in the deployment descriptor. A human tester can analyze 
changes in performance for different settings in the deplVment descriptor. However, 
test system 1 10 could be programmed to automatically edit me deployment descriptor of 
a bean by changing parameters affecting pooling or memoryVsage. Test system 110 
could then automatically gather and present data showing the rtnpact of a deployment 
descriptor on performance of an application. \ 



Even higher levels of automation could be achieved by testNsystem 110. For 



example, test system 110 might test the beans in an application and analyze the results of 
testing eac\ bean. Test system 110 might identify the bean or beans that reflect 
performance tsottlenecks (i.e. that exhibited unacceptable response times for the lowest 
numbers of simlidtaneous users). Then, test system 1 10 could run tests on those beans to 
Ind settings in tHe deployment descriptors that would balance the performance of the 
beans in the appl^tion (i.e. to adaptively adjust the settings in the deployment 
descriptors so that theVottleneck beans performed no worse than other beans.) 

It should also beVppreciated that computer technology is rapidly evolving and 
improved or enhanced versions of the hardware and software components making up the 
application under test and thVtest system are likely to become available. It should also 
be appreciated that the description of one device in a class is intended to be illustrative 
rather than limiting and that otheMevices within the same class might be substituted with 
ordinary skill in the art. For cxMe, the application under test was described in the 
context of a conventional applicatioii accessed through a client on a PC using a web 
browser as a graphical user interface, ft should be appreciated, though, that if the clients 
are intended to be household appliances with micro-controllers, a different interface 
might be readily substituted for the graphics user interface. 

Also, it was described that FIG. 11 shdws summary data of a test after execution 
is completed, ft will be appreciated, though, thk data on a test case execution might be 
displayed to a human tester while a test is in process. For example, the summary screen 
might contain a field that shows the percentage of the test case that is completed. This 
value would update as the test runs. Likewise, the ^alues for average and maximum 
response time could be updated as the data is gathered. 

Also, it was described that the objects being testedVe EJB components, which 
are written in the Java language. The same techniques Ve equally applicable to 
applications having components implemented in other lanVages. For example, 
applications written according to the COM standard might be written in Visual Basic and 
applications written for the CORBA standard might be written in Ch 
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Regardless of the specific language used, these standards are intended to allow 
eparately deveroped components to operate together. Thus, each must provide a 
'mechanism for otner applications, such as test system 1 10, to determine how to access the 
methods and properties of their components. However, there could be differences in the 
specific commands used to access components. 



In one embodimeht, code generator 212 is implemented in a way that will make it 
easy to modify for genera\ing test code for applications written in a different language. 
Specifically, code generato\ 212 stores intermediate results as a symbol table that is 
independent of the specific Isuiguage used to program the application under test. The 
symbol table lists methods ana properties for the component being tested. When to 
access these methods and what o^ta to use for a particular test and what kinds of data to 
record can be determined from thk information in the symbol table and input from the 
user. Thus, much of the functioningVf code generator 212 is independent of the specific 
language used to implement the application under test. 

In this way, the language specint aspects of code generator 212 are easily 
segregated and represent a relatively small part of the code generator 212. In particular, 
language specific information is needed to access the application under test to derive the 
information for the symbol table. Language specific information is also needed to format 
the generated client test code. But, it is intended\that these parts of code generator 212 
could be replaced to allow test system 1 10 to test applications written in other languages. 
Also, it is possible that test system 110 will contaim multiple versions of the language 
specific parts and the user could specify as an input the\anguage of the application under 
test. 

Therefore, the invention should be limited only by\he spirit and scope of the 
appended claims. 



Delete the paragraphs beginning at page 40. line 1 through page 55. line 35. 
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